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Abstract — — > 

Robot coordination and control systems for remote 
teleoperation applications are by necessity implemented * 
on distributed computers. Modeling and performance | 
analysis of these distributed robotic systems is difficult , i 
but important for economic system design. Perfor- \ 
mance analysis methods originally developed for con - 
ventional distributed computer systems are often unsat- 
isfactory for evaluating real-time systems. The paper 
introduces a formal model of distributed robotic control 
systems ; and a performance analysis method , based on 
scheduling theory, which can handle concurrent hard- 
real-time response specifications. Use of the method is 
illustrated by a case study of remote ieleoperation which 
assesses the effect of communication delays and the al- 
location of robot control functions on control system 
hardware requirements. 

1 Introduction 

As ambitious robotic applications are envisioned 
and more complex robot designs attempted, the need 
increases for efficient methods to evaluate their per- 
formance. Many of these new applications will be im- 
plemented on distributed computers. For instance, re- 
motely operated and multiple-robot applications are 
by their nature spatially distributed, and so necessi- 
tate a distributed real-time system for robot coordi- 
nation and control. The introduction of multiple pro- 
cessors, communication delays, and probabilistic per- 
formance of common-access communication channels 
in distributed systems complicates prediction of their 
real-time performance. 

Performance analysis methods for conventional dis- 
tributed systems employ one of three approaches: sim- 
ulation, stochastic models, or semantic models [7], 
These methods have complementary strengths and 
weaknesses; so, several methods may be needed to an- 
alyze all aspects of system performance. The char- 
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acteristics of these methods relevant to analyzing dis- 
tributed real-time systems are summarized in Figure 1. 

Simulation is arguably the most widely used ap- 
proach. In a simulation model, the actual operation 
of the system is duplicated in software at an abstract 
level of detail. The fidelity of the simulation depends 
upon accurately representing the structural properties 
of the system such as precedence of operations and 
contention for resources; and its timing properties such 
as execution times, communication latencies, and sen- 
sor polling delays. A simulation can produce a full 
probability distribution of system response times; and 
so, provide a complete characterization of soft- and 
hard-real-time performance. Thorough characteriza- 
tion comes at a price: a high level of detail is needed 
for good accuracy, but is computationally expensive. 
Also, complex systems have an extremely large num- 
ber of states that may necessitate excessively lengthy 
simulation duration to ensure that all states are exer- 
cised. For this reason, simulations are poor for prov- 
ing system correctness and global properties such as 
boundedness and freedom from deadlock. 

Stochastic models (e.g. Markov chains, queuing net- 
works, Petri nets) are also commonly used for perfor- 
mance analysis, particularly for evaluating communi- 
cation networks. In this approach, the system is ideal- 
ized as a finite set of discrete states with known prob- 
ability distributions for the transition rates between 
states. The model may be solved to estimate probabil- 
ity of each state as a function of time from which av- 
erage system performance may be derived. For simple 
systems an efficient, analytical solution is often pos- 
sible, and correctness and global properties may be 
determined. However, stochastic models of complex 
systems can be analytically intractable; requiring ap- 
proximation methods which may compromise fidelity 
and increase computation. 
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Semantic modeling is a less common approach to 
assess system performance that arises from computa- 
tional science theory of program correctness. In this 
approach, the logical and temporal relationship be- 
tween operations of the system are defined by process 
algebras or assertional calculi. Correctness and time- 
liness properties are then established by solving the 
model via a theorem prover. Semantic models are ef- 
fective for proving that timing specifications are satis- 
fied, but do not necessarily provide quantitative mea- 
sures of system performance. The downfall of semantic 
models is their computational complexity; verification 
is impractical for large systems. 

None of the three approaches described above is 
fully satisfactory for estimating performance of dis- 
tributed systems having hard-real-time response re- 
quirements. In a hard-real-time system, response times 
must never exceed specifications; and so, the system 
must be analyzed for worst-case performance. Simula- 
tion can produce estimates of worst-case performance, 
but at a high computational cost which becomes pro- 
hibitive when the system is designed for multiple con- 
current responses. Stochastic models give average re- 
sponse times only, and thus do not provide the infor- 
mation necessary to gauge performance of a hard-real- 
time system. Semantic models excel at proving cor- 
rectness and global properties, but are poor at quan- 
tifying response times. A fourth approach, based on 
scheduling theory, is proposed in this paper to specifi- 
cally address distributed hard-real-time systems. 

In the new performance analysis method, a for- 
mal model describes distributed real-time system or- 
ganization and its responses to external inputs. Sys- 
tem software is modeled as independent tasks that 
communicate by messages. Application of scheduling 
theory enables the calculation of guaranteed response 
times for task executions and message deliveries. The 
model provides a framework for formulating a con- 
straint satisfaction problem on processor and commu- 
nication channel schedules and on real-time require- 
ments whose solution defines system response times. 
The performance analysis problem may be solved to 
minimize weighted system response time or to mini- 
mize hardware cost while meeting response time re- 
quirements. The subsequent paper sections outline the 
system model, show the formulation of the constraint 
satisfaction problem, and then illustrate its use in an 
example. 


While this work has been motivated by the design 
of robot coordination and control systems, the perfor- 
mance analysis techniques are believed to have broader 
application to many distributed real-time systems. 


Method 

Provably 

Correct 

Real-Time Response 

Computing 

Limitations 

Soft 

Hard 

Expense 

Simulation 

Poor 

Good 

Good 

High 

high level of 
detail needed 
for fidelity 

Stochastic 

Models 

Fair 

Good 

Fair 

Moderate 

some problems 
intractable; 
approx, may lead 
to poor fidelity 

Semantic 

Models 

Excel. 

Poor 

Fair 

High 

v. difficult to 
develop model; 
large problems 
intractable 

Scheduling 

Theory 

(Good) 

Fair 

Good 

Low 

pessimistic for 
so ft- real- time; 


Figure 1: Performance Analysis Method Comparison 

2 Performance Analysis System Model 

Distributed computer systems are composed from 
multiple, independent processors connected by com- 
munication links. The characteristics of distributed 
systems can vary widely as the result of bandwidth 
and propagation delay of the interprocessor connec- 
tions. At the extremes are “tightly-coupled” multi- 
processor computers in which processors share a high- 
speed bus, and “loosely- coupled” multicomputer sys- 
tems which comprise separate computers connected by 
a network. Processor independence distinguishes dis- 
tributed systems from parallel computers in which pro- 
cessors typically are identical, and share data streams 
and/or instruction decoding. 

The proposed formal model can represent dis- 
tributed systems with arbitrary communication net- 
work topography; and so, can model the full range 
from multiprocessor to multicomputer system. In fact, 
in the model, a single node of a multicomputer network 
may be a complete multiprocessor. The new method is 
particularly useful for loosely-coupled systems, where 
access to communication channels as well as processor 
usage must be scheduled, since few analysis techniques 
are available for this class of distributed system. 

Because the independent computers of a distributed 
systems do not share physical memory, any data to 
be exchanged between processors must be transmitted 
across a communication link. The most common way 
to design distributed software that Accommodates this 
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restriction is to organize functions as independently- 
executing tasks that communicate via messages. This 
paradigm of tasks and messages is used in the formal 
model to define the system software, although the def- 
inition of a message has been generalized to include 
less-structured signals such as sensor inputs or control 
outputs. 

Some tasks must execute on specific processors; for 
instance, a sensor polling task must run on the proces- 
sor that is interfaced to the sensor hardware. However, 
in general, there are many choices of how to distribute 
software on the hardware. The actual assignment of 
tasks to processors has a strong influence on system 
performance; and so, must be specified for performance 
to be predicted. Optimal task assignment is an impor- 
tant design problem for distributed systems. We are 
currently experimenting with use of the new perfor- 
mance analysis method to guide task assignment [4]. 

Robotic systems, and indeed most real-time sys- 
tems, interact with their environment. Sensors gather 
data to characterize the environment. The control sys- 
tem monitors sensors to detect occurrence of specific 
conditions or events to which the system is designed to 
respond. And the system effects changes to the envi- 
ronment through its actuators; thus forming a closed- 
loop system. Also, in most systems, a human operator 
may intervene to modify goals or to initiate actions. 
Performance of robotic systems may be measured in 
many ways: accuracy, reliability, cost, etc. As we are 
primarily concerned with the computer system provid- 
ing robot coordination and control, performance will 
be defined as the end-to-end response time from when 
a condition can be sensed until a control signal is sent 
to actuators. Therefore, system response requirements 
are identified by input-output events and a response 
time specification. The requirement specifies a max- 
imum response time since we are dealing with hard 
real-time systems. 

From this description we see that four components 
are needed to fully describe a distributed real-time sys- 
tem: 

• software system model 

• hardware system model 

• assignment of tasks to processors 

• system response requirements 

In the definition of each model component, covered in 
the following subsections, we have attempted to de- 
scribe distributed real-time systems in terms that are 



Figure 2: Software Model of Teleoperation Example 


as similar as possible to how they are actually con- 
structed. While this tends to specialize the model, it 
has the benefit of providing a more natural represen- 
tation of a system implementation which, hopefully, 
improves ease of use and accuracy. 

2,1 Software System Model 

A distributed robotic application is typically con- 
structed from many, concurrent tasks that execute 
on multiple processors, and communicate by message 
passing between tasks, or between task and sensor or 
actuator. Each task corresponds to a software mod- 
ule, and the resulting system may be described by a 
directed graph with nodes corresponding to tasks and 
arcs representing messages. Each task is a separate 
software module that may execute periodically or upon 
demand (“aperiodic” or “event-driven”). This system 
level graph defines the topography of the communica- 
tions between tasks. 

Figure 2 shows a system level view of a simpli- 
fied control system for the teleoperated robot example 
that will be described in Section 4. The example em- 
ploys a hierarchical architecture loosely modeled on the 
NASA/NBS Standard Reference Model for Telerobot 
Control System Architecture (NASREM) [1]. System 
software is modeled with five periodic tasks and three 
event-driven tasks, which are shown as boxes in the 
figure. Periodic tasks are identified by their clock in- 
puts (circles). Input (sensors, keyboards) and output 
(actuators, displays) devices are represented by trian- 
gles whose orientation denote direction of data flow. 
Messages are shown as arrows from sending task to 
receiving task. A total of fourteen messages are trans- 
mitted between tasks in the example. 
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At a more detailed module level , each task is viewed 
as a finite state machine where task states or actions 
are nodes, and transitions between actions are directed 
arcs. The transition arcs are labeled with Real-Time 
Logic predicates [3] which define the condition causing 
the transition to occur. The purpose of the module 
level view is to define the response of an individual 
task to the input messages it receives. The finite state 
machine representation of the task allows a different 
computation time and different set of output messages 
to be defined for each input message. 

Each action node in the finite state machine repre- 
sents a deterministic sequence of operations that are 
delimited by a decision branch or a message transmis- 
sion/arrival. When a node is entered it executes for a 
fixed time interval and then optionally sends a message 
prior to blocking in that state or transitioning to an- 
other. Computation times are associated with actions, 
while transitions are instantaneous. The optional mes- 
sages produced by module actions correspond to the 
messages output from modules at the system level. 
Messages are identified only by type and bit length; 
the data values contained in a message are not consid- 
ered. 

Figure 3 shows the finite state machines for two 
tasks from the teleoperated robot example. The VI- 
SION PROCESSING task periodically acquires a camera 
image frame, transmits the frame as a video 1 message, 
processes the image to locate objects in the robot’s en- 
vironment, and then outputs the positions of the ob- 
jects in a OBJPOS message. Task processing is initiated 
when a CLK signal is received; and, when complete, the 
task returns to Idle Wait state to await the next signal. 
Figure 3b shows the finite state machine for the aperi- 
odic plan GENERATION task. This task is invoked by 
the arrival of a message rather that a clock signal; and 
contains two paths so that goal and error messages 
may be processed differently. Note that execution of 
the task is interrupted at Action 4 while it waits for 
requested data. Definition of periodic and aperiodic 
tasks are essentially the same at the module level — 
differing only by whether a clock signal or a message 
activates the task. 

2.2 Hardware System Model 

The purpose of the hardware system model is to 
describe how processors are interconnected, and the 
capabilities of processors and communication chan- 
nels. The principal capabilities of interest are processor 
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Figure 3: Finite State Machines for Example Modules 


speeds, and communications bandwidth and propaga- 
tion delay. 

Processor interconnections are represented by a 
hardware graph in which graph nodes correspond to 
processors, and graph arcs indicate the one-hop com- 
munication links between processors. Dedicated, uni- 
directional communication channels are shown as di- 
rected arcs; and shared communication channels (half- 
duplex or broadcast media) as sets of non-directed 
arcs. Any connection topography can be modeled in 
this way including loosely-coupled multicomputer net- 
works, bus-based multiprocessors, and combinations of 
the two. 

Figure 4a is a diagram of a multicomputer sys- 
tem with four single-processor workstations and a 4- 
processor multiprocessor connected by a local area net- 
work. One sensor and one actuator are interfaced to 
the multiprocessor. Figure 4b is the corresponding 
hardware graph. Note how the shared multicomputer 
LAN and the shared multiprocessor bus are expanded 
into sets of bi-directional links that fully interconnect 
all processors sharing each communication medium. 
There are no dedicated links in this example. 

2.3 Task Assignment 

The distribution of software modules onto computer 
hardware is described by first numbering all tasks and 
processors, and then defining an assignment function 
a which maps a task to a processor. Thus if task is 
assigned to processor pj then a(i) = j. This definition 
allows us to reference the hardware properties of the 
processor on which a task runs. 

A similar assignment function can be defined to ref- 
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Figure 4: Hardware System 



Model Example 


erence the communication hardware used by a mes- 
sage; thus if message is assigned to communication 
link lj then a(i) = j. Once tasks are assigned to pro- 
cessors the communication link over which a message 
travels is defined. Therefore the message assignment 
function can be derived from the task assignment func- 
tion plus the software and hardware system models; 
and so, we only need to specify the task assignment 
function. 

2.4 Response Requirements 

For this work, the principal performance measure 
is response time. System response time is defined as 
the time interval between occurrence of an external 
event and the system response. When sensor polling 
delays and actuator response times are factored out, 
the response time can be expressed as the time between 
an external input (sensor reading, operator command, 
etc.) and an external output (control signal, display 
update, etc.) of the control system. 

System response requirements specify the events to 
which the system must respond, the expected actions, 
and response time. Requirements correspond to the 
environmental constraints on the robotic system. We 
will consider only hard-real-time requirements in which 
a maximum response time is specified and must be 
satisfied. 

Most robotic systems will respond to many differ- 
ent events; and so, multiple response requirements will 
be defined. In hard-real-time applications, the sys- 
tem is expected to process concurrent events within 
their maximum response times under all load condi- 


tions. It is this requirement for concurrent responses 
that makes analysis of hard- real- time systems difficult. 
Contention for processors and communication channels 
will vary as the mix of concurrent events and their rela- 
tive overlap varies. For instance, it is difficult to ensure 
that sufficient simulations have been performed such 
that the worst-case combination of concurrent events 
is modeled. And, average response times obtained from 
stochastic models provide no information regarding re- 
sponse degradation under load. A key advantage of 
a scheduling theory-based approach is that its results 
hold for all phasings of task invocations, i.e. degree of 
overlap of concurrent events. 

3 Formulation of Performance Analysis 
Constraint Satisfaction Problem 

With the information contained in the system model 
described above, a constraint satisfaction problem can 
be formulated whose solution yields an estimate of sys- 
tem response times. Performance of the distributed 
robotic system is defined by a set of constraint equa- 
tions relating hardware, software, and response times. 
These equations are presented in the following subsec- 
tions. 

This system of equations is underconstrained; and 
so, a cost function is added to reduce the degrees of 
freedom. Different solution objectives can be achieved 
with different cost functions. In particular, the con- 
straint equations may be solved to yield minimum sys- 
tem response times for fixed hardware capacities, or to 
find minimum-cost hardware which can meet all sys- 
tem response time requirements. 

The problem is summarized as: 

• Minimize system response times or hardware cost 

• Subject to: 

1. Having a feasible schedule on every proces- 
sor and communication channel 

2. Meeting system response time requirements 

3. Satisfying bounds on individual task and 
message response times 

3.1 Cost Functions 

If no constraints are mutually exclusive then the 
constraint satisfaction problem can be solved. How- 
ever, since it is typically underconstrained, the prob- 
lem can have an infinite number of solutions. A cost 
function is included which introduces additional con- 
straints to ensure that only one solution is produced. 
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Through our choice of cost function we can direct the 
solution to achieve various design objectives. 

System response time is one possible cost func- 
tion. Since the system may have multiple responses, 
a weighted sum of response times is used to give a 
single- valued function. This allows us to emphasize 
one system response over another. A large penalty is 
assigned for exceeding a system response requirement, 
so all requirements are met before responses are further 
minimized. When this cost function is used, hardware 
capacities are held constant; hence, this form is useful 
for evaluating existing hardware. 

As an alternative, hardware capacities may be al- 
lowed to vary, and hardware cost used as the cost func- 
tion. The problem solution yields values for hardware 
capacities as well as system response times. This form 
of the constraint satisfaction problem is useful for eval- 
uating proposed hardware designs. The example per- 
formance analysis in Section 4 is formulated in this 
manner. 

3.2 Scheduling Constraints 

A principal distinction between performance analy- 
sis methods is how they handles resource contention. 
Analysis methods for real-time systems must be able 
to represent the order of internal system events so 
that resource contention can be modeled. Usually this 
means that the protocols for scheduling task execu- 
tions and message deliveries must be known. Sim- 
ulation methods use this information directly; while 
stochastic models represent resource contention prob- 
abilistically. The scheduling-based performance anal- 
ysis method presented here requires that a priority- 
based, preemptive scheduling protocol be employed for 
both processors and communication channels. Real- 
time operating systems typically implement such pro- 
tocols for processors; however, communication proto- 
cols supporting time-constrained messages are recent 
developments [9] [2], and are much less common. 

The reason the scheduling-based method is re- 
stricted to priority-based, preemptive protocols is that 
it depends on their predictable properties. With this 
class of scheduling protocol the execution time of the 
highest priority task is always known, and the worst- 
case execution times of lower priority tasks can be de- 
termined by assuming all higher priority tasks must 
execute first. In 1973, Liu and Layland [6] proved sev- 
eral properties of priority- based, preemptive schedul- 
ing protocols and introduced an analysis technique 


known as the rate monotomc scheduling algorithm. 
They established criteria for multiple tasks executing 
periodically on a single processor that, when satisfied, 
guarantee a schedule can be found in which all tasks 
meet their execution deadlines. They also showed that 
an optimal schedule is obtained by assigning priorities 
based on task periods — shortest period task has high- 
est priority. The original work on scheduling unipro- 
cessors has been extended to systems with aperiodic 
tasks and to shared communication channels, and is 
now referred to as generalized rate monotonic schedul- 
ing [5] [8]. 

In the proposed performance analysis method, 
scheduling theory criteria are used to identify the con- 
ditions under which a set of tasks [messages] can be 
scheduled such that they are guaranteed to meet ex- 
ecution [delivery] deadlines. These deadlines are then 
used as guaranteed response times. We have devel- 
oped a modified form of the generalized rate mono- 
tonic scheduling algorithm which applies to the robotic 
system model with event-driven tasks and real-time 
constraints. 2 This modified scheduling criterion gives 
the minimum speed S* of a processor [or communica- 
tion link] required to successfully schedule the tasks [or 
messages] assigned to it: 


S](C,f,6) 


max min 
{l<i<Af,} {rgSP,} 




It 


( 1 ) 

where C, r, and 0 are vectors of computation times, 
deadlines (guaranteed response times), and periods of 
tasks [messages] assigned to processor [link] j, respec- 
tively. N t is the number of assigned tasks [messages], 
and SPi is the set of critical scheduling points as de- 
fined by: 


SPi = {(i-l^+rj I 

lH* r j |i=i... .Lyjj} 

Note that computation times ( 7 , are normalized for a 
“standard” processor defined to have a relative speed 
of 1. Processor speed and 5* are expressed as relative 
speeds by ratioing to the standard processor. Messages 
and communication channels are treated in the same 
manner. 

2 Strictly speaking, since the technique uses deadlines 
rather than periods it should be referred to as deadline- 
monotonic scheduling. However, for clarity the more famil- 
iar term will be used. 
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The scheduling constraints require that the mini- 
mum relative speed 5* for a feasible schedule be less 
than or equal to the relative speed S of the processor 
or communication link: 

Sj{C> r, 0) < Sj for every processor and link j (2) 

The scheduling criterion defined by equation 1 es- 
sentially forms a ratio between demand for execution 
capacity (summation term) and available capacity (r). 
The ratio is checked at critical scheduling points which 
occur at deadlines and periods. Execution capacity de- 
mand is calculated for all tasks of priority j and higher 
priority tasks which may preempt it. The minimum ra- 
tio over all scheduling points reflects the lowest speed 
at which these tasks are schedulable for a given pri- 
ority. Finally, the ratio is checked for all priorities, 
and the worst case defines the relative speed needed to 
successfully schedule the assigned tasks or messages. 
3.3 System Response Time Constraints 

As defined in Section 2.4, system response require- 
ments are specified in terms of the external event which 
invokes a response, the expected system action, and 
response time. An external event detected by the sys- 
tem’s sensors will trigger a cascade of task executions 
and message transmissions. Many tasks may execute 
concurrently and multiple messages may contend for 
shared communication channels. The precedence of 
task executions and message transmissions associated 
with a particular event can be derived from the soft- 
ware model and is represented as a weighted directed 
acyclic graph called an event response graph. Graph 
arcs are weighted with task execution times and mes- 
sage delivery times, which are dependent on the un- 
derlying hardware capabilities. Since the graph is de- 
terministic, a critical path through the graph can be 
found that defines system response time for the specific 
event. 

As an example, consider the response of the sys- 
tem from Figure 2 to a high-level command input by 
an operator. The command is received by the INTER- 
FACE manager task which interprets the command 
and then transmits a goal message to PLAN genera- 
tion. In subsequent processing steps data is obtained 
from the WORLD MODEL, a plan created and sent to 
PLAN EXECUTION, and so on until the system response 
to the high-level command is produced at the robot. 
The complete sequence of task executions and message 
transmissions is shown in the event response graph in 
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Figure 5: An Event Response Graph 


Figure 5. This simple example has a linear critical 
path; but, in general, the critical path may contain 
parallel legs. Control system response time is calcu- 
lated by summing guaranteed task execution times of 
the seven tasks in the graph including polling delays 
at the periodic tasks, plus guaranteed message deliv- 
ery times of the six messages including propagation 
and switching delays, plus communication time associ- 
ated with sensor input or actuator output. Note that 
the PLAN GENERATION task appears twice in the ex- 
ample event response graph. The first invocation of 
PLAN GENERATION is in response to a goal message, 
and the second in response to a data message. Execu- 
tion times for PLAN generation are different in each 
instance as defined in the module level finite state ma- 
chine description of the task (see Section 2.1). 

The fact that event response graphs must, be deter- 
ministic does not prevent us from modeling probabilis- 
tic events such as failures. In these cases, an event 
response graph would be developed to represent the 
processing that occurs for each possible outcome; and, 
potentially, each outcome could have a separate hard- 
real-time response requirement. If a system is required 
to meet a response time specification even in the pres- 
ence of failure then only the more restrictive situation 
would have to be modeled — probably the case in- 
cluding the additional processing to accommodate fail- 
ure. Alternatively, a less demanding response time re- 
quirement could be defined for failure processing which 
would yield a less costly control system design. This 
type of analysis enables us to study tradeoffs between 
system reliability and cost. 

An event response graph is constructed for each sys- 
tem response having a time requirement. Since guar- 
anteed task execution times and guaranteed message 
delivery times are solution variables of the problem, 
system response time can be determined by summing 
the variables corresponding to the weights on the event 
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response graph. The constraint equations are formed 
by requiring that system response time must be less 
than its requirement for each response: 

y; r, < Rk for all responses k (3) 

ti t rr\i€CP k 


where r, is the guaranteed response time of task or 
message i, CPk is the set of tasks and messages on 
the critical path for response fc, and /Z* is the system 
response time requirement. 

3.4 Task/Message Response Time Bounds 

Response time for an individual task or message is 
bounded. Response time can not be less than the time 
required to execute the task or transmit the message, 
and is not allowed to be greater than its period. This 
upper bound results from a restriction that at most 
one invocation of a task is allowed to execute at a time. 
For aperiodic tasks, a parameter analogous to period 
is specified to be the minimum interval between exe- 
cutions. These bounds place the following constraints 
on guaranteed response times: 


Cj 

S c(i) 


< r i < 


for all tasks and messages (4) 


where S a (t) is the relative speed of the processor to 
which ti or is assigned (recall that a is the assign- 
ment function), Oi is the period or minimum interar- 
rival time of the task or message, and the other terms 
retain their earlier definitions. 

Task/message response time bounds can be repre- 
sented as simple variable bounds for constraint satis- 
faction problems with constant processor and commu- 
nication channel speeds since all of the terms in the 
calculation of the lower and upper bounds would be 
known and constant. However, if hardware speeds are 
solution variables, then the lower bounds must be in- 
corporated as nonlinear constraint equations. 

3.5 Solving Constraint Equations 

In summary, to analyze the performance of a dis- 
tributed robotic system we first define the system by 
the model outlined in Section 2; then form the sys- 
tem of constraints from equations 2, 3, and 4. The 
constraint satisfaction problem is solved to minimize 
the cost function, i.e. to minimize weighted system 
response time, or to minimize hardware cost. The so- 
lution provides times for all system responses, guaran- 
teed response times for task executions and message 


deliveries, and processor and communication channel 
speeds. 

A nonlinear programming method is needed to solve 
the constraint equations. Unfortunately, although 
equation 1 is continuous it is not smooth. Therefore, 
nonlinear methods such as sequential quadratic pro- 
gramming and others that require smooth derivative 
information can not be used. The system of constraints 
has been successfully solved with a successive linear 
programming approach. We believe that this approach 
works because the partial derivatives of equation 1 are 
piece wise-linear. 

4 Example Use of Analysis Method 

This section presents an example use of the new 
performance analysis method for design of the con- 
trol computer system of a teleoperated robot. The 
minimum-cost hardware formulation will be used to se- 
lect capacities of processors and communication links 
for various design conditions of communication delay 
and task assignment. 

Control software is organized in a “NASREM-like” 
architecture as seen earlier in Figure 2. The stan- 
dard components of sensory processing, world model- 
ing, task decomposition, and operator interface are all 
included; however, only the task decomposition func- 
tions are modeled in sufficient detail to show a hier- 
archical organization. The eight tasks comprising the 
system are listed in Table 1 with their relative compu- 
tation times and execution periods. Note that the task 
decomposition functions of plan generation, plan 
EXECUTION, TRAJECTORY GENERATION, and BASIC 
control form a hierarchy with execution period dif- 
fering by an order of magnitude between levels. Param- 
eters for the messages transmitted among the tasks are 
listed in Table 2. 


Task 

Comp Time, ms 

Period, ms 

Basic Control 

4 

10 

Traj. Generation 

30 

100 

Plan Execution 

50 

1000 

Plan Generation 

2000 

- 

World Model 

50 

- 

Vision Processing 

170 

100 

Video Relay 

0.1 

100 

Interface Manager 

10 

10 


Table 1: Task Parameters for Example 


Figure 6 shows the control system hardware for 
the teleoperation example. It includes a local proces- 
sor at ground station, a remote processor at an or- 
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Message 

Length, kbits 

Period, ms 

GOAL 

7.8 

- 

PLAN 

7.8 

- 

PATH 

3.7 

1000 

HCINP 

0.4 

10 

SETPT 

1.1 

100 

POS 

0.4 

10 

OBJPOS 

7.3 

100 

UPDATE 

7.8 

100 

STATUS 

7.8 

1000 

ERROR 

0.1 

- 

REQ 

0.8 

- 

DATA 

7.8 

- 

VIDEOl 

25 

100 

VIDE02 

25 

100 


Table 2: Message Parameters for Example 


bital facility, and control and vidpp processors on the 
robot to support dedicated control and video prepro- 
cessing functions. Three communication channels con- 
nect these processors: unidirectional uplink and dnlmk 
channels between ground and orbit, and a radio net- 
work, designated met , for communications between 
robot processors and the orbital facility. The nomi- 
nal assignment of tasks to processors locates VISION 
PROCESSING on vidpp , BASIC CONTROL on control , IN- 
TERFACE MANAGER on local, and all remaining tasks 
on the remote processor. 


Camera — j_ 

Robot 




UPUNK 


REMOTE 

DNLINK 

LOCAL 


1> CHtplay 
— ^ Keyboard 


Hand 
Con trotter 


Figure 6: Hardware for Teleoperation Example 


Five time-critical responses are specified, and serve 
as the hard-real-time system response time require- 
ments. They are listed on Table 3. The control system 
must display information about the work site in three 
forms: live video at 10 frames/second, a reconstruc- 
tion of the world model updated by object recognition, 
and a model showing robot position. The system must 
guarantee that data from each of the three sources is 
delivered to the INTERFACE MANAGER in 2.4 seconds 
(2400 ms) so that it can be fused into a consistent dis- 
play. An operator controls the robot either indirectly 
through high-level commands, or directly via a hand 
controller. The system is expected to respond to high- 
level commands in 9600 ms, and hand controller input 
in 1200 ms. As covered in section 3.3, an event re- 
sponse graph is constructed for each system response 


requirement to identify the tasks and messages invoked 
to process each response. 


Description 

Mnemonic 

Requirement 

Display Live Video 

LIVE-DSP 

2400 ms 

Display World Model 

WM.DSP 

2400 ms 

Display Robot Position 

ROELDSP 

2400 ms 

Respond to HL Command 

CMD.RSP 

9600 ms 

Respond to Hand Controller 

HC.RSP 

1200 ms 


Table 3: System Response Requirements for Example 


Relative costs for processors and communication 
channels were modeled with a power function: cost = 
multiplier x speed exponent . Ground facilities were as- 
signed a 1.0 multiplier, orbital facilities were assumed 
to have an order of magnitude higher multiplier, and 
processors on the robot have an additional factor of 
two premium. Exponents of 2.0 for ground and 1.5 
for orbital facilities were used. The cost model should 
have an additional additive factor; but the SLP solu- 
tion method being used cannot support it. 

4*1 Effect of Communication Delays 

In the first design study, the effect of communica- 
tion delay on processor and communication channel 
capacity is examined. The new performance analysis 
method is used to find the minimum hardware needed 
to guarantee that system response time requirements 
are achieved. Communication delays of 100, 500, and 
1000 ms are studied. These values represent propa- 
gation and switching delays only; and so, may appear 
low compared to customarily quoted values which in- 
clude scheduling delays due to traffic contention. The 
performance analysis method computes the scheduling 
delays. 

Table 4 summarizes key results. As required, all 
system responses meet requirements. At 100 and 500 
ms delays the high-level command response is limiting; 
whereas, the world model display response limits at 
1000 ms delay. 

Most non-limiting responses differ between cases by 
an amount equal to the difference in communication 
delay. This is a consequence of the solution method 
which focuses on the requirements that constrain hard- 
ware speed while essentially ignoring responses not at 
a limit. Requirements for non-limiting responses could 
be lowered to the reported values without affecting 
hardware speeds. 

Faster processors and communication channels are 
needed as communication delay increases. At delays of 







500 ms and lower, the more expensive control and vidpp 
processors are at minimum capacity needed to meet 
execution periods of their assigned tasks. A modest 
increase in remote processor speed is sufficient to ac- 
commodate a communication delay of 500 vs. 100 ms. 
However, for the 1000 ms delay, all processor speeds 
must be higher in order to meet system response re- 
quirements. 

Total relative hardware cost differs little between 
100 and 500 ms cases. However, the cost of the video 
preprocessor dominates total cost, thereby masking the 
10% difference in cost of all other components between 
the cases. The design for 1000 ms delay is significantly 
more costly: 246% total and 137% system excluding 
vidpp costs relative to the 500 ms design. 

The point of this analysis is not to draw conclu- 
sions regarding an admittedly over-simplified teleop- 
erated robot application, but rather to demonstrate a 
possible use for the new performance analysis method. 
It is feasible to guide key design decisions, in this case 
by examining tradeoffs between control system costs 
and communication switching infrastructure, through 
use of real-time system analysis. 


Case # 

1 

2 

3 

Comm Delays, ms 

100 

500 

1000 

System Responses, ms | 

- LIVE-DSP 

360 

760 

1210 

- WM.DSP 

1760 

2060 

2400 

- ROB.DSP 

1580 

1880 

2270 

- CMD.RSP 

9600 

9600 

9380 

- HC.RSP 

280 

670 

1160 

[ Processor Capacity j 

- control 

0.40 

0.40 

0.65 

- vidpp 

1.70 

1.70 

3.55 

- remote 

1.22 

1.36 

1.52 

- local 

1.00 

1.00 

1.12 

| Comm Link Capacity | 

- uplink 

0.99 

0.98 

1.03 

- dnlink 

0.99 

0.98 

0.99 

- met 

0.10 

0.11 

0.11 

[ Relative Cost | 

- system ex vidpp 

0.90 

1.00 

1.37 

- vidpp 

1.00 

1.00 

3.02 

- total system 

0.97 

1.00 

2.46 


Table 4: Effect of Communication Delays 


4.2 Effect of Task Assignment 

Another use of the new performance analysis 
method is illustrated in this section as a design study 
evaluating the effect of task assignment on hardware 
cost. Communication delays are fixed at 500 ms for 


all cases. In the base case, PLAN GENERATION, PLAN 
execution, and trajectory generation tasks exe- 
cute on the remote processor, and the BASIC CONTROL 
task on the control processor. 

The effect of moving first PLAN generation and 
then PLAN EXECUTION to the local processor was mod- 
eled with the results shown in Table 5. Cost savings 
can be obtained by shifting computing load from ex- 
pensive orbital processors to lower cost ground com- 
puters while still meeting response time requirements. 
For this simplified example, the analysis suggests that 
the savings may be substantial, and may motivate 
further study to assess the impact on other mission- 
critical factors such as the reliability and safety impli- 
cations of remote computing. 

Migrating dedicated processing at the robot to the 
somewhat less expensive computing available at Space 
Station may also be cost effective. Table 6 summa- 
rizes modeling results for moving the control task 
to the remote processor. This reassignment eliminates 
the control processor which is replaced by simpler hard- 
ware to receive control signals from met ; and remote 
processor capacity is correspondingly increased. Com- 
munication latency of the control signals are on the 
order of 0.3-0. 4 ms which should be acceptable. The 
full benefit of relocating control functions may not be 
achievable since some at-robot processing capability is 
required for safety functions which have not been mod- 
eled. 

5 Future Work 

Currently we are working to improving the efficiency 
of the performance analysis method; in particular, to 
increase robustness of the successive LP solution ap- 
proach and to decrease computation time. The prin- 
cipal motivation for improving solution efficiency is so 
that the performance analysis method may be embed- 
ded in a genetic algorithm with the objective of find- 
ing near-optimal task assignments. If successful, this 
would provide a powerful tool for designing distributed 
real-time systems in which software module allocations 
and hardware are optimized concurrently. 

Other activities are aimed at demonstrating the ca- 
pabilities of the performance analysis method on a va- 
riety of robotic systems, and directly comparing re- 
sults to those obtained from simulation and stochastic 
models. Theoretical and experimental verification of 
performance analysis tools will provide an important 
contribution to the field of robotics, and will form the 
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Case # 

2 

4 

5 

Comm Delays, ms 

500 

500 

500 

Task Assignments 


remote 

local 

local 


remote 

remote 

local 


remote 

remote 

remote 


control 

control 

control 

1 System Responses, ms | 


760 

820 

900 



2400 

1970 


3 


1790 




9600 


670 


680 

J Processor Capacity [ 

- control 

0.40 



- vidpp 

1.70 

1.70 


- remote 

1.36 


0.81 

- local 

1.00 

| 1.32 

1.35 

Comm Link Capacity | 

- uplink 


0.92 


- dnlink 

0.98 

0.90 


- met 

0.11 

0.11 


| Relative Cost j 

j - system ex vidpp | 100 

r 0.72 



Table 5: Effect of Shifting Tasks to Local Proc 


basis for more efficient development of new robotics 
applications in the future. 
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